Peer-reviewed journal of AI-agents: a challenging LLM competition

Lecture given by Isabelle Guyon (Google Research, ChaLearn, and Université Paris-Saclay), Kent Rachmat, Khuong Thanh Gia Hieu (LISN/INRIA/CNRS, Université Paris-Saclay)

Abstract: Significant recent advancements have occurred in the field of question answering, particularly within LLM-powered chatbots and augmented search engines like ChatGPT, Bard, the new Bing, and similar platforms. These advancements offer promising prospects for accelerating the acquisition of academic knowledge across various disciplines, including science, social sciences, education, and other fields of study. In academia, progress is typically facilitated through a peer-reviewed literature system. This has inspired us to emulate this process using AI agents.

In our envisioned scenario, AI agents would emulate a peer-reviewed journal. Human contributors would assume roles as editors or meta-reviewers, providing prompts (call-for-papers) to AI authors and selecting papers for publication based on the evaluations from AI reviewers, as well as their own judgment. Acceptance or rejection of papers would serve as a teaching signal for both AI authors and AI reviewers, motivating them to continuously enhance their skills.

To begin working towards this ambitious objective, we are organizing a challenge for the AutoML-conf'23 conference. Participants will be invited to submit AI agents capable of acting as authors and reviewers. For this challenge, we will focus on generating systematic surveys or overview papers.

To generate prompts for the challenge, we reverse-engineered numerous papers from various fields indexed in Semantic Scholars, including Computer Science, Medicine, Chemistry, Biology, Materials Science, Physics, Geology, Psychology, Art, History, Geography, Sociology, Business, Political Science, Economics, Philosophy, Mathematics, Engineering, Environmental Science, Education, Law, and Linguistics. These papers served as a basis for creating prompts such as: "Write a systematic survey or overview examining the impact of social media on mental health. This paper should explore current research on the correlation between social media usage and mental health outcomes, encompassing areas such as depression, anxiety, and self-esteem."

Furthermore, we have developed a baseline AI author and AI reviewer as a starting point. During the conference, we will present the competition design and our initial results based on the baseline models. We will also provide instructions on how to submit your first entry to the competition.

Acknowledgments: The support of INRIA, Google Research and ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022 and TAILOR EU Horizon 2020 grant 952215 are gratefully acknowledged.

Biography: Isabelle Guyon recently joined Google Research as a director of research. She is also professor of artificial intelligence at Université Paris-Saclay (Orsay). Her areas of expertise include computer vision, bioinformatics, and power systems. She is best known for being a co-inventor of Support Vector Machines. Her recent interests are in automated machine learning, meta-learning, data-centric AI, and large language model. She has been a strong promoter of challenges and benchmarks, and is president of ChaLearn, a non-profit dedicated to organizing machine learning challenges. She is community lead of Codalab competitions, a challenge platform used both in academia and industry. She co-organized the “Challenges in Machine Learning Workshop” @ NeurIPS between 2014 and 2019, launched the "NeurIPS challenge track" in 2017 while she was general chair, and pushed the creation of the "NeurIPS datasets and benchmark track" in 2021, as a NeurIPS board member.

Mots clés : ai agents chatgpt llm-powered chatbots peer-review